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MEDIA COMPOSER INCLUDING POINTER— BASED 
DISPLAY OF SEQUENTIALLY STORED SAMPLES 

Cross R eference to Related Application 

This application is a continuation-in-part of 
application Serial No. 07/866,829, filed April 10, 
1992, entitled "IMPROVED MEDIA COMPOSER" , herein 
incorporated by reference. 

Field of the Invention 

The invention relates to the display of 
sequentially stored samples under the control of a 
pointer, and- more particularly, to the use of a 
mouse as a jog or shuttle control for computer-based 
video and audio composition. 

Background of the Invention 

It is known to provide on a media player, such 
as a video tape recorder, a jog and/or shuttle 
knob. A shuttle knob allows the operator of the 
media player to adjust the speed of the player in 
either a forward or reverse direction. A jog 
control allows the user to move in forward and 
reverse directions through the medium on a still 
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frame by frame basis. Examples of prior art jog and 
shuttle controls include levers , knobs, concentric 
knobs , and knobs that change function when pressed 
upon axially or when another switch is pressed. 

A common system is a three-button system, in 
which the user places three of his or her fingers on 
three buttons. The outer two fingers control the 
forward and reverse shuttle operations, which are 
much like fast forward scan and rewind scan buttons 
on a video tape recorder, except that multiple 
presses on the outer buttons in shuttle mode result 
in proportional increases in shuttling speed. For 
example, three presses will cause shuttling to take 
place at triple the shuttling speed. The central 
button is a stop button, and when it is held down, 
the outer buttons act as forward and reverse, jog 
controls. 

In computer-based media composition systems, a 
visual representation of a shuttle or jog control 
may be displayed and manipulated with a mouse. For 
example, an image of a slider or a film strip may be 
presented to the user. The user may then click on 
the slider with the mouse and drag the slider to 
perform either shuttle or jog operations. For 
example, the Quantel Harry system, available from 
Quantel Systems, Inc. of England, uses this type of 
metaphor. Generally, however, these systems require 
the user to be watching the screen so that they may 
locate the mouse pointer in a proper position on the 
slider . 
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Summary of the Invention 

In general, the invention features selecting 
samples for presentation on an output device, such 
as a display or speaker, from a sequence of stored 
media samples, such as audio or video information. 
Position information is received from a pointing 
device, such as a mouse, and translated into 
direction and magnitude information. A second 
sample is then retrieved based on this position and 
magnitude information. This method may be used to 
implement jog or shuttle controls for a media 
composer, which may be provided with simulated 
"inertia" for ease of use. 

A media composer according to the invention has 
the advantage of convenience and efficiency for the 
user. The user may perform composition operations 
using the pointing device and, without removing his 
or her hand from the pointing device, move to the 
next location on the medium to be composed. The 
user may also do so without looking at a control 
screen at all; he or she need only look at the 
material to be composed. These capabilities are 
provided inexpensively in a commonly-available 
pointing device, which may already be a part of the 
user's composing system. Also, the added inertia in 
both jog and shuttle modes, and playback speed limit 
in shuttle mode may prevent the user from getting 
lost in the material to be composed. 
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Brief Description of the Drawings 

For a better understanding of the present 
invention, together with other and further objects, 
advantages and capabilities thereof, reference is 
made to the accompanying drawings, which are 
incorporated herein by reference and in which: 

FIG, 1 presents a media composing workstation 
portion, of a media composing system according to the 
invention; and 

FIG, 2 is a system block diagram of the media 
composing system of FIG, 1. 

Description, of the Preferred Embodiment 

Referring to FIG. 1, an exemplary media 
composition workstation according to the invention 
includes a computer output system 12 and computer 
input system 14, The output system may include one 
or more computer monitors 16 and one or more 
speakers 18* The input device may include a 
keyboard 20 and a pointing device 22, such as a 
mouse with one or more buttons 32. The display 
presented to the user on the monitor may be divided 
into a window 24 for variable speed playback, 
on-screen scrub controls 26, a timeline display 28, 
and a timeline position indicator 30. 

Referring to FIG. 2, a media composing system 40 
according to the invention may also comprise a 
mouse/scrub control module 42, a player module 44, a 
timeline module 46, and storage 48. The mouse/scrub 
control module is connected to the keyboard via a 
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data path 50 and to the pointing device by another 
data path 52. The player module 44 is connected to 
the mouse/scrub control module via a further data 
path 54. 

The player module is also connected to receive 
information from the storage 48 via a bus 56, which 
may comprise several data lines 64, 66 , 68. These 
data lines may be dedicated to different 
synchronized media channels, such as video and audio 
channels. The bus 56 is further routed to the 
timeline module 46, which also receives data from 
the player module via a data path 58. The player 
module may provide information to the moving picture 
window 24 via a picture information path 60, and may 
provide synchronized sound information to the 
speaker via a sound information data path 62. The 
timeline module supplies information to the timeline 
display 28 via a timeline information path 70. 

It is observed that the routing of data and 
information may differ from the block diagram in an 
actual implementation. For example, the moving 
picture window and timeline display may be displayed 
on the same monitor and therefore share a data 
line. The pointer data might also be routed through 
the keyboard on its way to the mouse/scrub module or 
the keyboard might not be needed at all. 
Furthermore, the invention herein is applicable to 
composition operations on sound alone, and therefore 
the displays may not be necessary. Alternatively, 
the operations may be performed on images alone, 
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making speakers unnecessary. 

The pointing device 22 may be one of several 
general purpose pointing devices available for 
computers . It may be a mouse supporting one or more 
buttons 32, or it may be a trackball , a light pen, a 
digitizing pad, touch pad, touch screen or the 
like. These types of pointing devices are commonly 
used to position a cursor on a screen of a computer 
system, or to receive position dependent data, such 
as that generated in a mianual digitizing operation, 

A system according to the invention may be 
developed using a variety of types of computer 
systems. In one embodiment, an Apple Macintosh and 
the M C" computer language were used, but naturally 
the invention should not be read to be limited to 
this particular platform. The mouse/scrub control 
module, player module, and timeline module may be 
implemented in software and/or hardware and may or 
may not be contained within the computer system. A 
software package entitled "Avid Media Composer", 
available from Avid Technology, Inc. of Tewksbury, 
Massachusetts includes these features. The storage 
may include a high-speed, high capacity internal 
magnetic hard disk drive. For further information 
on computer-based media storage and playback of 
video and audio information, see Peters et al-. , 
"VIDEO AND AUDIO TRANSMISSION SYSTEM AND METHOD", 
U.S. patent No. 5,045,940, issued September 3, 1991, 
herein incorporated by reference. 

Storage 48 contains one or more sequences of 
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image and/or sound samples, and may also contain a 
sequence of composition information, such as editing 
and special effect information. The images may be 
stored in compressed form, allowing more images to 
be stored in a given amount of space. The sequences 
of images and sound are generally related in time 
much like the tracks of a multiple track type 
recording. For example, the storage may provide 
multiple logical connections including one or more 
for picture information 64, one or more for sound 
information 66, and one or more for composition 
information 68. 

Referring to FIGS, l and 2, in" composing 
material, the user interacts with the workstation 
portion of the system 10. Composition operations 
may include editing operations, special effects 
generation, dubbing operations, or the like. The 
composition to be worked on is displayed in an 
outline format in the timeline display. This 
outline format may include waveforms for sound 
information, or a series of representative pictures 
for image information such as video or film 
sequences. If a user desires to perform a composing 
operation on. a particular scene he, or she may move 
the pointer 34 on the screen using the mouse and 
click on the portion of the timeline which shows the 
representative scene indicating the location at 
which the edit is to be performed. This is a 
relatively rough operation as computer displays will 
generally only allow display of a relatively low 
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number of images on a timeline at a meaningful size. 
Once the user has located a rough location for 

the edit using the timeline position indicator, he 

or she may perform more precise jog and shuttle 

operations. To do so, the operator may use the 

pointing device to click on the on-screen controls 

26 or may press a key 21 on the keyboard 20. This 

operation puts the workstation in shuttle mode. In 

this mode, when the user moves the mouse to the 

left, the workstation begins moving through the 

stored sequence of images at a rate determined by 

the distance through which the user has moved the 

mouse. This corresponds to a reverse shuttle 

operation, which may analogized to scan rewinding a 
video tape at a variable rate, with the rate 

dependent on the mouse displacement . Conversely, 
moving the mouse to the right will cause the 

sequence of images to be moved through in a forward 
direction in a forward shuttle operation. 

The system 40 performs this shuttle operation by 
first obtaining position information from the 
pointing device through the data path 52. This 
operation may be performed by directly accessing the 
hardware associated with the pointing device, or by 
retrieving a coordinate stored by the computer's 
operating system. This mouse position information 
is then translated by the mouse/scrub control module 
into direction and magnitude information, which is 
transferred to the player module. 

The magnitude information may be represented by P 
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the X-coordinate of the current position of the 
mouse subtracted from the X-coordinate of the 
position of the mouse when the mode was entered, 
with this difference being multiplied by a scaling 
constant. The sign of this difference may be used 
as the direction information. The value of the 
constant will determine how sensitive the shuttle 
operation will be. 

The direction and magnitude information can be 
transformed in this operation into numerical values 
which can be directly applied to the player module 
as commands. The direction information represents 
the direction of playback for the player module, and 
the magnitude information represents the speed of 
playback. These relayed commands directly cause the 
player module to display the sequences of images 
and/or sounds on the display at the appropriate rate. 

In the shuttle mode, the system limits the 
maximum forward or reverse speed to three-times the 
sound speed (the sound speed is the speed at which a 
video sequence is played where the sound is played 
at the proper speed) . This speed limiting feature - 
prevents the user from getting lost by moving the 
mouse at a high rate of speed, such as by 
accidentally knocking the mouse across the desk. 

Furthermore, in shuttle mode, "inertia" is added 
to the operation of the control. Rather than 
supplying information about mouse position changes 
directly to the player, changes in mouse position 
are moderated over a short period. In one 
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embodiment this is done by responding to changes in 
mouse position by providing a. smaller corresponding 
change in speed to the player module and gradually 
increasing, that change in speed to the full change 
in speed indicated by the mouse. For example , if 
the user were to double the distance of the mouse 
from its starting position,- the mouse scrub control 
module might convey a 50% increase to the player 
module, fallowed closely by a 75% increase, and then 
a 90% increase, and finally a 100% increase- It has 
been found that this inertia enhances the ease of 
use and efficiency of use of the apparatus . 

In shuttle mode, the system also has a small 
threshold value below which no shuttle operation 
takes place. This prevents the player from playing 
back at extremely slow speeds which are of little 
use and are sometimes found to be uncomfortable to 
the user. Values on the order of a 1% threshold 
have been found to be satisfactory. 

The user may also use a jog mode for the 
workstation. Jog mode is similarly controlled by 
the position of the mouse, but advances from still 
frame to still frame within the sequence of images, 
rather than adjusting the speed of playback of the 
images. In jog mode, the distance the mouse travels 
from its starting point relates to the offset within 
the sequence of frames between the current position 
and the position to be displayed. 

To enter jog mode from shuttle mode, the user 
may simply press and hold the mouse button 32. This 
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stops the action in the display window and 
subsequent motion of the mouse will constitute 
jogging operations. Other methods for entering the 
jogging mode are possible, such as keyboard keys 
21. Similarly, the user may exit the jog mode by 
one of these methods, or by "double clicking on the 
mouse button 32. Releasing the mouse button while 
in jog mode will return the user to shuttle mode, 
and with the current mouse position as the starting 
position at zero play speed. 

In the jog mode, the mouse scrub control module 
receives position information from the pointing 
device and translates it into a different kind of 
direction and magnitude information. The mouse 
scrub control module constantly monitors the mouse 
and performs velocity calculations based on the 
position information of the mouse over time. The 
resulting velocity information may be supplied 
directly to the player module, after it is adjusted 
by a constant. By making the speed of the player 
proportional to that of the mouse, the player module 
will track the mouse position. 

Inertia may also be added to the jog operations 
in a manner similar to that disclosed above in 
connection with the shuttle mode. This is 
particularly useful in connection with sound 
information, as it makes sound playback more regular 
during a jog operation. 

It is noted that the timeline moves during both 
jog and shuttle operation and that this provides 



WO 93/21595 



PCT/US93/03356 



- 12 - 



feedback to the user in these operations. This 
feedback enhances the ease of use of the workstation 
operations and prevents the user from getting lost. 
Also, the user or system designer may adjust the 
constants determining the sensitivity of the jog and 
shuttle controls and the inertia, threshold and 
maximum speeds in the shuttle mode. This allows, the 
user or designer to customize the "feel" of the 
workstation. 

While there have been shown and described what 
are at present considered the preferred embodiments 
of the present invention, it will be obvious to 
those skilled in the art that various changes and 
modifications may be made therein without departing 
from the scope of the invention as defined by the 
appended claims. 



WO 93/21595 



- 13 - 



PCT/US93/03356 



CLAIMS 

-What is claimed is: 

1. A method of selecting samples for 
presentation on an output device from a sequence of 
stored media samples, comprising the steps of: 

presenting a first sample retrieved from a 
current position in the sequence on the output 
device, 

receiving position information from a pointing 
device, 

translating the received position information 
into direction and magnitude information, 

retrieving a second sample stored at a position 
in the sequence that is offset from the current 
position by an amount related to the magnitude 
information and in the direction indicated by the 
direction information, and 

presenting the second sample on the output 
device. 

2. The method of claim l wherein the samples in 
the sequence are each stored images, wherein the 
output device comprises a display device, and 
wherein the steps of presenting the first and second 
samples respectively include displaying first and 
second one of the images on the display device. 



3. The method of claim. l wherein the samples i 
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the sequence are each stored sound samples, wherein 
the output device comprises an audio device and 
wherein the steps of presenting the first and second 
samples respectively each include playing the first 
and second audio samples on the audio device. 

4 .. The method of claim 1 wherein the step of 
translating the received position information 
translates the position indicated by the pointing 
device into a playback speed for successively 
playing back samples in the sequence to implement a 
shuttle control. 

5. The method of claim 4 wherein said step of 
translating further includes the step of only 
translating the received position into a playback 
speed if the received position falls above a 
predetermined threshold. 

6 . The method of claim 4 f urther including the 
step of limiting the playback speed. 

7. The method of claim l wherein the step of 
translating the received position information 
translates the position indicated by the pointing 
device into a sample position within the sequence to 
implement a jog control. 

8. The method of claim 7 wherein the step of 
translating moderates the effect of changes in 
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velocity of the pointing device on the rate of 
playback of the samples. 

9. The method of claim 7 wherein the step of 
translating includes determining the velocity of the 
pointing device and translating it into a playback 
speed for successively playing back samples in the 
sequence. 

10. The method of claim 7 wherein the step of 
translating moderates the effect of changes in 
velocity of the pointing device on the rate of 
playback of the samples. 

11. The method of claim 1 wherein the pointing 
device is a mouse, 

12. The method of claim 1 further including, 
before the step of presenting the first sample, 
determining the current position using a coarse 
timeline control. 

13. The method of claim 1 further including the 
steps of, after an exit command, using the 
information from the pointing device to perform 
other position-sensitive operations. 



14. A media composing system comprising: 
storage for a sequence of samples, 
a pointing device, 
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a control module responsive to the pointing 
device, for translating position information from 
the pointing device into player module commands 
including direction and magnitude information, 

a player module responsive to the control module 
and the storage „ for playing back selected ones of 
the samples in response to the player module 
commands r and 

an output device responsive to the player module 
for presenting the samples to a user. 
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